Features

Editor features

  • UNICODE support.
  • Fully customizable lexer supports RegExp, so editor is suitable for any plane text format.
  • Text folding, any determined text range or user range may be collapsed.
  • "Line Changed" color indicator on gutter
  • Normal, column-oriented and line selection, clipboard operations.
  • Export to RTF, HTML file with syntax highlighting. Copy selection to clipboard in RTF.
  • User ranges allows custom lexer independent highlighting and text folding.
  • Stack-based markers like in CodeRush
  • Animation allows to mark out some position in text.
  • No last letter cutting (for italic fonts). This problem is present even in Delphi 2005 editor.
  • The SyncEdit feature from CodeGear BDS2007 lets developers simultaneously edit identical identifiers in code. As changes are made to the first identifier, the same change is performed automatically on the other identifiers.

Highlighting features

  • Highlighting styles can use any font.
  • Text border styles.
  • Syntax highlighting associated with simple lexical elements (tokens).
  • Text range (sequence of tokens) highlighting. Any detected text range can have style that will be merged with token styles.
  • Nested lexers highlighting. Combining several lexers in one using "Sub Lexer" rules.
  • Highlighting of the user ranges (over lexer defined highlighting).

  • Active highlighting depended on caret position. Highlights valid blocks (for example, current procedure, brackets, "begin" ... "end" blocks ...).
  • Line highlighting associated with text ranges, caret position, gutter objects or custom.
  • Powerful style merging (token style, text range style, dynamic style, …)
  • Text partition by means of line separators like in Visual Basic IDE.
  • Hierarchical text representation using block staples.

  • Hiding parts of the text using Hidden attribute of the style.

Using Mouse

Right button click Show popup menu;
Left button click Move caret to the mouse position and start normal selection;
Alt + Left click Move caret to the mouse position and start column selection;
Alt + Ctrl + Left click Move caret to the mouse position and start line selction.
If during start selection Shift key have been pressed, block from current caret position to mouse click position is selected.

Left button click over selection block - start text dragging. Dragged text will be moved to new position. To copy selection using drag&drop press Control key during draggng.

You can perform drag&drop on block staples. Press Control key and drag block staple. This operation will indent/unindent lines that are coupled by the block staple.


History

    v2.04 (12.12.07)
  • (f)WordWrap of russian text
  • (f)Possibly fixed 'Cannot open Clipboard' error
  • (f)Search settings (Ctrl-R panel) saved now
  • (f)Error when cursor over dir in TC quick view
  • (f)Position of Setup dialog off the screen
  • (f)Display of Setup->Editor->TabMode
  • (f)Wrong staples look on print/zoom
  • (f)Open deleted file from MRU
  • (f)Ask to Save when close Tab by DblClick
  • (+)Move line up/down (Alt-Up/Dn)
  • (+)Duplicate line (Ctrl-D)
  • (+)Reload next file in QuickView without reload of plugin
  • (+)Select lines by click on gutter
  • (+)Hint when ACP list browsing
  • (+)ACP filter behavior in options
  • (+)Gutter width calculation when number of lines changes
  • (+)Next/Prev/Clear SearchMarks hotkeys
  • (+)Reload File when changed outside (Options->Misc)
    v2.03 (20.11.07)
  • (f)AV when open Options/About
  • (f)Lost edit focus when activate syn2.exe
  • (f)Wrong paint of toolbars/scrollbars
  • (f)Options->Misc->History numbers now saved
  • (f)AV in SDI mode, when last windows close
  • (f)State saves now for all Tabs when multiple closed
  • (f)Crush when save to ReadOnly file
  • (f)Fixed bug with find current word (Ctrl-Shift-Up/Dn)
  • (+)Hotkeys for next/prev tab in MDI mode (Ctrl-Tab / Ctrl-Shift-Tab)
  • (+)Default Codefolding toggle in Options->Editor
  • (+)Zoom hotkeys(Ctrl-Num+ / Ctrl-Num-)
  • (+)Close Tab by DblClick
  • (+)In ReadOnly mode Space=PgDn, Cursor - scrolls text
    v2.02 (10.11.07)
  • (f)Hotkeys list sorted
  • (f)PHP string rule updated
  • (f)Detectstring fixed to open blank files
  • (f)AV at CheckSynchronize possible fix
  • (+)Loader Syn2.exe (have MultiTab mode always enabled)
  • (+)Chm context help for curent word/selection by F1
  • (+)Forward 1..4,N,P keys to lister when in ReadOnly or Alt pressed
  • (+)'Copy URL' command in url context menu
  • (+)Help updated (Statusbar fields, encoding selection)
  • (+)colored hints when mouse over function from ACP
  • (+)MidleClick on Tabs free space - creates new blank Tab
    v2.01 (3.11.07)
  • First public release


File structure

Some notes about files used by plugin. Directory structure looks like:

Cfg\Search.dat - Latest search/replace queries
Cfg\State.dat - Latest files state (cursor position, top line, bookmarks, etc)
Cfg\syn.ini - Configurable thru Setup dialog options
Cfg\htm.int - internal HTML tag/values autocompletion (Works only if current lexer at cursor named as 'HTML')
HL\*.acp - Files used for AutoCompletePopup (Ctrl-Space), Parameters help (Ctrl-Shift-Space) and Functions descriptions (Hold mouse over function). Name of these files must be same as name of lexer. (For example 'php.acp' for 'PHP' lexer)
HL\*.lcf - Lexer files for single languages. Note that these files don't load to plugin! If you want to use some language, you need to open 'Manage Library' subitem in lexers menu. And add needed lexers to library.
HL\LexLib.LXL - Main lexer library file. All lexers used by plugin loaded only from this file! For faster loading better to keep only needed lexers, so seldom used lexers removed from library and saved as *.lcf files.
Syn2.wlx - main plugin dll file.
Syn2.exe - external loader for Syn2.wlx. You can use it as text editor, assign to TotalCommander F4 hotkey, or to some filetype in Windows...


Lexer configuration

Overview

Lexer - is a configuration of lexis analysing component. It is a resource, it is not a separate code implementation.
No programming is required to create support for new text file format.

Lexis analysis is based on the syntax rules. There are three type of rules:

  • Token rules is intended to extract tokens from the text.
  • Token block rules (block rules) is intended to find text ranges (group of tokens), line separators and key words dection.
  • Sub lexer rules is intended to define ragions where nested lexers will act.

Styles is used to define text drawing params. It is possible multiple stylies will be applied to a portion of text. Style merging procedure specifies sequence in which styles are merged into canvas params.

Lexer library - is a collection of lexers. It allows to automate file extension managing for selecting appropriate lexer, constructing filter for open|save dialogs, open/save lexers into single file.
If you use nested lexers it's recomendned to include master and slave lexers into lexer library.


Syntax of Regular Expressions

Introduction

Regular Expressions are a widely-used method of specifying patterns of text to search for. Special metacharacters allow You to specify, for instance, that a particular string You are looking for occurs at the beginning or end of a line, or contains n recurrences of a certain character.

Regular expressions look ugly for novices, but really they are very simple (well, usually simple ;) ), handly and powerfull tool.

Simple matches

Any single character matches itself, unless it is a metacharacter with a special meaning described below.

A series of characters matches that series of characters in the target string, so the pattern "bluh" would match "bluh'' in the target string. Quite simple, eh ?

You can cause characters that normally function as metacharacters or escape sequences to be interpreted literally by 'escaping' them by preceding them with a backslash "\", for instance: metacharacter "^" match beginning of string, but "\^" match character "^", "\\" match "\" and so on.

Examples:
foobar matchs string 'foobar'
\^FooBarPtr matchs '^FooBarPtr'

Escape sequences

Characters may be specified using a escape sequences syntax much like that used in C and Perl: "\n'' matches a newline, "\t'' a tab, etc. More generally, \xnn, where nn is a string of hexadecimal digits, matches the character whose ASCII value is nn. If You need wide (Unicode) character code, You can use '\x{nnnn}', where 'nnnn' - one or more hexadecimal digits.

\xnn char with hex code nn
\x{nnnn} char with hex code nnnn (one byte for plain text and two bytes for Unicode)
\t tab (HT/TAB), same as \x09
\n newline (NL), same as \x0a
\r car.return (CR), same as \x0d
\f form feed (FF), same as \x0c
\a alarm (bell) (BEL), same as \x07
\e escape (ESC), same as \x1b

Examples:
foo\x20bar matchs 'foo bar' (note space in the middle)
\tfoobar matchs 'foobar' predefined by tab

Character classes

You can specify a character class, by enclosing a list of characters in [], which will match any one character from the list.

If the first character after the "['' is "^'', the class matches any character not in the list.

Examples:
foob[aeiou]r finds strings 'foobar', 'foober' etc. but not 'foobbr', 'foobcr' etc.
foob[^aeiou]r find strings 'foobbr', 'foobcr' etc. but not 'foobar', 'foober' etc.

Within a list, the "-'' character is used to specify a range, so that a-z represents all characters between "a'' and "z'', inclusive.

If You want "-'' itself to be a member of a class, put it at the start or end of the list, or escape it with a backslash. If You want ']' you may place it at the start of list or escape it with a backslash.

Examples:
[-az] matchs 'a', 'z' and '-'
[az-] matchs 'a', 'z' and '-'
[a\-z] matchs 'a', 'z' and '-'
[a-z] matchs all twenty six small characters from 'a' to 'z'
[\n-\x0D] matchs any of #10,#11,#12,#13.
[\d-t] matchs any digit, '-' or 't'.
[]-a] matchs any char from ']'..'a'.

Metacharacters

Metacharacters are special characters which are the essence of Regular Expressions. There are different types of metacharacters, described below.

Metacharacters - line separators

^ start of line
$ end of line
\A start of text
\Z end of text
. any character in line

Examples:
^foobar matchs string 'foobar' only if it's at the beginning of line
foobar$ matchs string 'foobar' only if it's at the end of line
^foobar$ matchs string 'foobar' only if it's the only string in line
foob.r matchs strings like 'foobar', 'foobbr', 'foob1r' and so on

The "^" metacharacter by default is only guaranteed to match at the beginning of the input string/text, the "$" metacharacter only at the end. Embedded line separators will not be matched by "^'' or "$''.
You may, however, wish to treat a string as a multi-line buffer, such that the "^'' will match after any line separator within the string, and "$'' will match before any line separator. You can do this by switching On the modifier /m.
The \A and \Z are just like "^'' and "$'', except that they won't match multiple times when the modifier /m is used, while "^'' and "$'' will match at every internal line separator.

The ".'' metacharacter by default matches any character, but if You switch Off the modifier /s, then '.' won't match embedded line separators.

TecRegExpr works with line separators as recommended at www.unicode.org ( http://www.unicode.org/unicode/reports/tr18/ ):

"^" is at the beginning of a input string, and, if modifier /m is On, also immediately following any occurrence of \x0D\x0A or \x0A or \x0D (if You are using Unicode version of TecRegExpr, then also \x2028 or \x2029 or \x0B or \x0C or \x85). Note that there is no empty line within the sequence \x0D\x0A.

"$" is at the end of a input string, and, if modifier /m is On, also immediately preceding any occurrence of \x0D\x0A or \x0A or \x0D (if You are using Unicode version of TecRegExpr, then also \x2028 or \x2029 or \x0B or \x0C or \x85). Note that there is no empty line within the sequence \x0D\x0A.

"." matchs any character, but if You switch Off modifier /s then "." doesn't match \x0D\x0A and \x0A and \x0D (if You are using Unicode version of TecRegExpr, then also \x2028 and \x2029 and \x0B and \x0C and \x85).

Note that "^.*$" (an empty line pattern) doesnot match the empty string within the sequence \x0D\x0A, but matchs the empty string within the sequence \x0A\x0D.

Metacharacters - predefined classes

\w an alphanumeric character (including "_")
\W a nonalphanumeric
\d a numeric character
\D a non-numeric
\s any space (same as [ \t\n\r\f])
\S a non space

You may use \w, \d and \s within custom character classes.

Examples:
foob\dr matchs strings like 'foob1r', ''foob6r' and so on but not 'foobar', 'foobbr' and so on
foob[\w\s]r matchs strings like 'foobar', 'foob r', 'foobbr' and so on but not 'foob1r', 'foob=r' and so on

Metacharacters - word boundaries

\b Match a word boundary
\B Match a non-(word boundary)

A word boundary (\b) is a spot between two characters that has a \w on one side of it and a \W on the other side of it (in either order), counting the imaginary characters off the beginning and end of the string as matching a \W.

Metacharacters - iterators

Any item of a regular expression may be followed by another type of metacharacters - iterators. Using this metacharacters You can specify number of occurences of previous character, metacharacter or subexpression.

* zero or more ("greedy"), similar to {0,}
+ one or more ("greedy"), similar to {1,}
? zero or one ("greedy"), similar to {0,1}
{n} exactly n times ("greedy")
{n,} at least n times ("greedy")
{n,m} at least n but not more than m times ("greedy")
*? zero or more ("non-greedy"), similar to {0,}?
+? one or more ("non-greedy"), similar to {1,}?
?? zero or one ("non-greedy"), similar to {0,1}?
{n}? exactly n times ("non-greedy")
{n,}? at least n times ("non-greedy")
{n,m}? at least n but not more than m times ("non-greedy")

So, digits in curly brackets of the form {n,m}, specify the minimum number of times to match the item n and the maximum m. The form {n} is equivalent to {n,n} and matches exactly n times. The form {n,} matches n or more times. There is no limit to the size of n or m, but large numbers will chew up more memory and slow down r.e. execution.

If a curly bracket occurs in any other context, it is treated as a regular character.

Examples:
foob.*r matchs strings like 'foobar', 'foobalkjdflkj9r' and 'foobr'
foob.+r matchs strings like 'foobar', 'foobalkjdflkj9r' but not 'foobr'
foob.?r matchs strings like 'foobar', 'foobbr' and 'foobr' but not 'foobalkj9r'
fooba{2}r matchs the string 'foobaar'
fooba{2,}r matchs strings like 'foobaar', 'foobaaar', 'foobaaaar' etc.
fooba{2,3}r matchs strings like 'foobaar', or 'foobaaar' but not 'foobaaaar'

A little explanation about "greediness". "Greedy" takes as many as possible, "non-greedy" takes as few as possible. For example, 'b+' and 'b*' applied to string 'abbbbc' return 'bbbb', 'b+?' returns 'b', 'b*?' returns empty string, 'b{2,3}?' returns 'bb', 'b{2,3}' returns 'bbb'.

You can switch all iterators into "non-greedy" mode (see the modifier /g).

Metacharacters - alternatives

You can specify a series of alternatives for a pattern using "|'' to separate them, so that fee|fie|foe will match any of "fee'', "fie'', or "foe'' in the target string (as would f(e|i|o)e). The first alternative includes everything from the last pattern delimiter ("('', "['', or the beginning of the pattern) up to the first "|'', and the last alternative contains everything from the last "|'' to the next pattern delimiter. For this reason, it's common practice to include alternatives in parentheses, to minimize confusion about where they start and end.
Alternatives are tried from left to right, so the first alternative found for which the entire expression matches, is the one that is chosen. This means that alternatives are not necessarily greedy. For example: when matching foo|foot against "barefoot'', only the "foo'' part will match, as that is the first alternative tried, and it successfully matches the target string. (This might not seem important, but it is important when you are capturing matched text using parentheses.)
Also remember that "|'' is interpreted as a literal within square brackets, so if You write [fee|fie|foe] You're really only matching [feio|].

Examples:
foo(bar|foo) matchs strings 'foobar' or 'foofoo'.

Metacharacters - subexpressions

The bracketing construct ( ... ) may also be used for define r.e. subexpressions

Subexpressions are numbered based on the left to right order of their opening parenthesis.
First subexpression has number '1' (whole r.e. match has number '0' - You can substitute it in TecRegExpr.Substitute as '$0' or '$&').

Examples:
(foobar){8,10} matchs strings which contain 8, 9 or 10 instances of the 'foobar'
foob([0-9]|a+)r matchs 'foob0r', 'foob1r' , 'foobar', 'foobaar', 'foobaar' etc.

Metacharacters - backreferences

Metacharacters \1 through \9 are interpreted as backreferences. \<n> matches previously matched subexpression #<n>.

Examples:
(.)\1+ matchs 'aaaa' and 'cc'.
(.+)\1+ also match 'abab' and '123123'
(['"]?)(\d+)\1 matchs '"13" (in double quotes), or '4' (in single quotes) or 77 (without quotes) etc

Modifiers

Modifiers are for changing behaviour of TecRegExpr.

There are many ways to set up modifiers.
Any of these modifiers may be embedded within the regular expression itself using the (?...) construct.

i
Do case-insensitive pattern matching (using installed in you system locale settings).
m
Treat string as multiple lines. That is, change "^'' and "$'' from matching at only the very start or end of the string to the start or end of any line anywhere within the string.
s
Treat string as single line. That is, change ".'' to match any character whatsoever, even a line separators, which it normally would not match.
g
Non standard modifier. Switching it Off You'll switch all following operators into non-greedy mode (by default this modifier is On). So, if modifier /g is Off then '+' works as '+?', '*' as '*?' and so on
x
Extend your pattern's legibility by permitting whitespace and comments (see explanation below).
r
Non-standard modifier. If is set then range а-я additional include russian letter 'ё', А-Я additional include 'Ё', and а-Я include all russian symbols.

The modifier /x itself needs a little more explanation. It tells the TecRegExpr to ignore whitespace that is neither backslashed nor within a character class. You can use this to break up your regular expression into (slightly) more readable parts. The # character is also treated as a metacharacter introducing a comment, for example:

(
(abc) # comment 1
| # You can use spaces to format r.e. - TecRegExpr ignores it
(efg) # comment 2
)

This also means that if you want real whitespace or # characters in the pattern (outside a character class, where they are unaffected by /x), that you'll either have to escape them or encode them using octal or hex escapes. Taken together, these features go a long way towards making regular expressions text more readable.

Perl extensions

(?imsxr-imsxr)
You may use it into r.e. for modifying modifiers by the fly. If this construction inlined into subexpression, then it effects only into this subexpression

Examples:
(?i)Saint-Petersburg matchs 'Saint-petersburg' and 'Saint-Petersburg'
(?i)Saint-(?-i)Petersburg matchs 'Saint-Petersburg' but not 'Saint-petersburg'
(?i)(Saint-)?Petersburg matchs 'Saint-petersburg' and 'saint-petersburg'
((?i)Saint-)?Petersburg matchs 'saint-Petersburg', but not 'saint-petersburg'

(?#text)
A comment, the text is ignored. Note that TecRegExpr closes the comment as soon as it sees a ")", so there is no way to put a literal ")" in the comment.


Lexer configuration files

There are two file extensions used to store lexer resources:

*.LCF Single lexer configuration
*.LXL Lexer library

All this files have "Delphi Form Resource" text format.

To manage this files you may use "Manage Library" dialog. This dialog contains commands: save/load lexer library (*.LXL), import/export single lexer (*.LCF). Library already contains lexers for common filetypes. You can add more lexers from "\HL" folder to library by importing them.


"Syntax Lexer" dialog

This dialog is intended for customizing lexer. It includes:
· Lexer common properties...
· Text styles.
· Token rules
· Block rules
· Sub lexer rules
· Code templates.
· Sample text.

To show this dialog you must click "Customize Lexer" button in main menu (lamp button)

Button commands.

Export Opens save dialog to save lexer configuration to the file (*.LCF).
Import Opens open dialog to load lexer configuration from file (*.LCF).
Apply Applies changes.
OK Save changes.
Cancel Cancel changes.

Preview text

At the bottom of the "Syntax Lexer" dialog editor with the sample code is placed. Any change of lexer configuration effects on the sample code. This preview text also is used in the "Syntax Highlighting" dialog that is intended to change only styles.

Common page

Lexer name - name of the lexer. Usually it is a name of file format (language).
File extensions - for autodetect lexer by file extensions
Token type names are associated with the integer value (line number). They are used to simplify token type assignment to the token rules and selecting a set of token types for conditions of block rules. Token type is integer value in th range [0..31].
Default Style - default style which first is applied to the editor canvas. For default style you may specify font and background color. If default style is not specified, editor Font and Color properties are used.
Selection mark style is a style for selected block in the syntax memo. If no style is selected to draw selection in the editor default foreground and background are used (clHighlightText and clHighlight).
Search mark style is a style to mark out search results in the syntax memo. If no style is selected to draw search results in the editor default foreground and background are used (clWhite and clBlack).
Current Line Style - style that is used to draw current line (line with the caret). If you don't want highlight current line select style type "Only Background" and color of background "None".
Default Style - default style which first is applied to the editor canvas. For default style you may specify font and background color.
Skip spaces while parsingcheck box specifies how to process space characters (space, HT/TAB, CR, NL, FF). If it is checked all space characters will be skipped before token rules checking, otherwise they will not be skipped.
Do not skip end of line - specifies whether line breaks should be parsed.
Restart analysis from line start - if it is ON parser results are cleared starting from start of line where text was changed, otherwise they are cleared after position of text changing.
Always synchronized block analysis - specifies whether block analysis should be always performed with text parsing, otherwise analyzer tries to perform block analysis after whole text was parsed.
Disable full refresh if text size is greater - specifies threshold after which full refresh (when text is changed) of sytactical obects will not be performed. If this value is 0 (by default), full refreshing is disabled. Full refreshing takes more time, but sometimes it may be useful for some parser's rules(for example, parser expression " ".*" " will not work correctly without full refresh).

Code templates page

At this page you can create, delete or change code templates .
If check box at the left side of the template in the list view is checked, template supports advanced syntax like in CodeRush, otherwise it supports simple syntax like in Borland's IDE.
In avanced templates you may use: <caret> - specifies caret position (must be single), <paste> - inserts clipboard text, <marker> - drops stack-based marker.
In simple templates you may specify only caret position using "|" character.
Available code templates are listed by name with a short description. While working in the Syntax Memo, press Ctrl+J to display the code templates defined. Click a template name to display the code that will be entered in your file when that template is selected. Code displayed in the code window can be edited. Double-click on a template to insert it into your code.

Rules and style are described in the next topics:
Common for styles and rules.
Styles ("Styles" page).
Token rules ("Parser" page).
Block rules ("Rules" page).
Nested lexers ("Sub lexers" page).


Common for Styles and Rules

All styles and rules is items derived from TSyntCollectionItem of corresponding collections, so working with list is common for them.

Each item in list have unique name and index.
Edit box under list allows to change name of the item.
You can use drag&drop operations to change order of items.

You can disable any item. Disabled items displayed with gray color. Disabled items will not be used in syntax analysis algorithm (rules) or will not be applied to the editor canvas (styles).

Commands:
1. Move item up one step.
2. Move item down one step.
3. Create new item.
4. Import items from other lexer. Open "Import Styles/Rules" dialog.
5. Disable/enable item.
6. Delete item.
7. Delete all items.

Parent restrictions for rules (TRuleCollectionItem)

All rule classes are derived from TRuleCollectionItem. In this class special conditions for rule enabling are defined.
You can enable rule only within (out) some block.

Parent block - block rule of opened text range (when rule is being checked) that is verified for presence.
Strict parent block - flag that specifies that parent block must be immediate parent for th rule.
Not a parent - flag that specifies that "parent block" is not a parent of the rule.

Example:

implementation

const y: integer = 5;

function A;
var x: integer;
begin
case B of
0: begin
x := y;
end;
end;
end;

 start of "implementation" range

token detection for "const"

start of "function" range
token detection for "var"
start of "begin" range
start of "case" range
start of "begin" range (second)

end of "begin" range (second)
end of "case" range
end of "begin" range, end of "function" range

For the detection "var" range "implementation" is a parent, but not a strint parent, range "function" is parent and strict parent.
For "function" range detecting you can specify parent "implementation". If you set strict parent flag, nested functions will not be detected. Such restrictions will work only in units (where "implementation" section is present). Or you may specify that "function" range have not a "interface" parent (set flag "not a parent") in this case function will be detected not only in units but in programm file or in scripts.

Always enabled - flag that specifies that this rule will be active not only in the current lexer but in the nested lexers too.
For example, in pascal function we have "asm" range with reference to nested "Assembler" lexer. We can specify that "comment" token rule is always enabled. In this case in the assembler block pascal comments will be highlighted correctly, otherwise this rule will be inaccessible in assembler block.

Style - reference to style item that is associated with the rule. Style affects differently for each type of rules, so it will be described below.


Name formatting (for block rules)

Name format string is used for formatting Name, Group name and Collapsed text icon of the text range.

In format string all entries of kind: %sINT, %eINT, %psINT, %peINT will be replaced with the token, index of which will be calculated on the assumption of the format expression.

Letters specifies the reference index.
%s - start of current text range
%e - end of current text range
%ps - start of parent text range
%pe - end of parent text range

There may be several entries of "p" metacharacter. For example, %pps - means start of parent of parent text range.

Also after these specifiers you may use "L" or "Z"
L - defines string from line start to the token (including this token);
Z - defines string from the token to the end of line (including this token);
For example:
%SL2 - defines string from line start to second token from text range start.

Token index will be calculated as reference index minus Index (INT).

Extension of name formatting syntax

Syntax
%(S|E)P*(L|Z)?[0-9]+

is expanded to
%(S|E)P*([\[]<token>[\]]<offset>?)?

where <token> is a specific token that is "searched from the specified starting point (S for first token in the range , or E for the last token) towards the respective range end (up- or downwards). The search-direction is kept in the variable "rngdir" which is set in the "S" , "E" decision.

example: line(s) is/are ".... for x = 1 to 12 do ...... end ; ..."
0 1 2 3 4 5 6 ... 27 28

range-start = "for",
range-end = "end"
then "...%s[to] ..." will skip forward to the token "to" (with index 4).
The token values are searched on a "asis" basis, there is no case-insensitivity option yet.
A "numeric number following the token value will define an <offset> relative to the found token.
For this clause, the variable "idx" is not set by taking the static numeric value as in "...%s2 ..." , instead the "found token index" is kept.

For "%S..." the search starts at idx=0 up to max 28. ---> rngdir = +1;
For "%E..." the search starts at idx=28 downto min 0. ---> rngdir = -1;

The options L or Z introduced in V2.35 will not combine with the new (range) specifying options --> somebody else may find a use for such extended ranges.
Notes: Avoid to search for tokens that can occur at multiple places (for example a ";" between statements).

The above syntax is simple as it allows to identify the
block-start-tokens "for x = 1 to 12 do"
block-body anything after block-start tokens up to
block-end-tokens "end ;"

but many syntax formats do not trivially support this separation.

The current implementation does not provide the information where "block-start", "block-body" and "block-end" are beginning/ending.
A "%B0..." for the "block-body" portion and a "ignore block-body tokens" option may be nice !?

b) any such clause (either absolute or given by token value) can "start a token range" by additionally specifying:

%(S|E)...~(S|E)P*[0-9]+
or
%(S|E)...~(S|E)P*([\[]<token>[\]]<offset>?)?
or
%(S|E)...~[\[]<token>[\]]

The first form uses the static index specification to define the end-range:
"%s0~s3" results in "for x = 1" (tokens 0, 1, ... 3)

The 2nd form uses the new syntax to "search for an end-token beginning at the starting range index (idx) up- or down-wards.
"%s0~s[do]" results in "for x = 1 to 12 do" (tokens 0, 1, ... 6) if a search is not satisfied, the complete range up to "e0" is taken. Because of the same "S", the search starts with "TagStr[idx]" ...

"s0~e[do]" results in the same string, but starts at the final "end" of the block and scanning downwards.
Caution: This may produce WRONG results if nested loops are scanned !
I could not find a valid representation of "range-start" token-streams, the range-body alone and/or the range-end token-stream.
Such information may be helpful to better display blocks and/or collapse display of the "block-body" alone.

The 3rd form is an abbreviation where the S/E indicators are taken to be identical as the starting point
S1~[do]1" results in "x = 1 to 12" (tokens 1, 2, ... 5)

The <offset> "1" will here skip back by 1 from the found token "do". The range-end is kept in the variable "to_idx".

The "token-value" to search for can not be whitespace #00..#20. Leading and trailing whitespace withing the "...[vvvvvvv] ..." enclosed by [ and ] characters sequence is removed before searching. the "vvvvvv" can contain escaped characters like "... [\]] ..." to allow "[" and/or "]" to be part of the value. The \r, \n, \f ...escapes are not supported here.

The token accumulation simply (?) takes all tokens from "idx" ... "to_idx" and builds a string by appending all tokens with ONE " " (blank) as separating delimiter. There is no process to keep the original token positions within the source line(s) and any whitepace including cr/lf's there. This may be an addition but I currently do not see a need for it.

c) "ranges as specified above may accumulate many tokens and it may be desirable to "limit" the result string.
This can be done by using another operand syntax

%(S|E)...~[0-9]+
or
%(S|E)...~(S|E)([\[]<token>[\]]<offset>?)?~[0-9]+
or
%(S|E)...~[\[]<token>[\]]~[0-9]+

In all three forms the "~" is immediately followed by a numeric value which is interpreted as "maximum number of tokens in the substituted string", if the range takes MORE than this maximum The value is internally kept in the variable "rngmax" below. When the result string is accumulated (taking all tokens between "idx" up- resp. down-to "to_idx") the number of appended tokens can not go beyond "rngmax". If this happens the result will be created in the form "t-1 t-2 -- t-max ..." with the ellipsis string " ..." appended.

Hans L. Werschner, Oct '07

Example:

function TMyClass.MyFunction ...

we have rule:
0| .
1| [any Identifier]
2| function

reference start of text range will point to ".", so
format string
"%s2 => %s1.%s-1"
results
"function => TMyClass.MyFunction"


Token rules (parser)

Common rule properties are described in "Common for Styles and Rules" part.

Token is an text element. Token have start and end position in text. Tokens can not intersect each other.
They are defined by the parsing procedure, after which token array is created. Token array is sequential, i.e. end position of token will be less start position of the next token.

Token rules are checked sequentially. After first successful checking of rule loop is broken.

Token rules are intended for detecting tokens. You can control token rules in the page "Parser" of the "Syntax Lexer" dialog.

Properties of token rule:

Token type - integer value that will be assigned to all tokens detected by means of this rule. It's required only if you want use this tokens in the block detection algorithm. To simplify token type assignment "Token type names" is used.

Token style - style that will be applied to all tokens detected by means of this rule.

Rule expression - regular expression for detecting token. Regular expression is applied to current analysis position. If it match current position is incremented ont the length of found token and token rules loop is broken. See "Syntax of Regular Expressions" to get more information.

Default modifiers for regular expression: (?imxr-s), i.e. ignore case, multiline (^, $ - start, end of line), supports comments, full russian char set, any metacharecter "." (dot) does not include end of line "\n".

Rule expression is most robust operation in lexer configuration. Below several example are given.

Identifier

[a-z_]\w*

first character is literal other is word charecter (including digits). If first character is literal rule text match rule.

String

'.*?('|$)

from the single quote to the next single quote or to the end of line. Multiple any char is "non-greedy" to check next condition ('|$) before incrementing position. ('|$) means either single quote or end of line.

Float const

#with exp. dot is optional
\d+ \.? \d+ e [\+\-]? \d+ |
#without exp. dot is required
\d+ \. \d+

\d+ - means at least one digit.
\.? - means optional dot
[\+\-]? - means optional "+" or "-".

Integer const

\d+

Note: integer const must be after float const because of any float will be treated as integer until dot or exponent symbols.

Single line comment

//.*

Any charecter from // to end of line. Metacharacter dot does not include end of line, so it is optional place $ end of line metacharacter at the end of expression.

Multiline comment

(?s)\{.*?(\}|\Z)

We turn on modifier "s" to include end of line in metacharecter "." (dot). It will select text from symbol "{" to the symbol "}" or to the end of text. If we skip end of text, i.e. ( (?s)\{.*?\} ) it will cause that comment is not detected.

Multiline define

(?-i)\#define
(.*\\\s*\n)* # lines with line folding
.* # last line

This is an example of complex expression for proprocessor directive "define".
\#define - means that token starts with #define
(.*\\\s*\n)* - means multiple any character until symbol "\", then it skips spaces, includes end of line. All this expression can be repeated some times (or not once).
.* - means all characters until end of line.


Rules

Block rules are applied to the array of tokens. Block rules are checked after each token detection.

Block types

Block type specifies action after successful conditions check.

There are four rule types:

  1. Tag detector - sets style of token (with Identifier index)
  2. Line separator - creates line separator
  3. Range start - creates text range
  4. Range end - closes text range.

Indexing in block rules

Some properties of block rule use indexes. All indexes are back offset from the indexing reference point.

There two case of reference point:

1. Start of conditions (index 0 corresponds to the first checked token).
2. End of conditions (index 0 corresponds to the last checked token).

Example:

Rule "class derived" of LexLib.Pascal:
0. EQUAL <identifier>
1. EQUAL ) <symbol>
2. SKIP <symbol, identifier>
3. EQUAL ( <symbol>
4. EQUAL class <identifier>
5. EQUAL = <symbol>

TComponent = class ( TPersistent , IInterface , IInterfaceComponentReference )
Conditions start 9 8 7 6 5 4 3 2 1 0
Conditions end 1 0 -1 -2 -3 -4 -5 -6 -7 -8

Due to there is SKIP condition in this rule, we do not know length of token sequence that satisfies this rule. To specify indexing reference point use Relative to end of condition. flag. When it is True indexing reference point - conditions end, otherwise indexing reference point - conditions start. In this example we must set this flag to True.

Identifier index - back offset of identifier token. In example above identifier index may be 1 (token "TComponent") or -1 (token "class").
The default value of the Identifier Index is 0. It is not necessary to specify this property.

Block offset - back offset of text range start/end. It is used only for "Range start/end" rules.

Grammar rule - name of grammar rule which is used instead of conditions sequence to test current sequence of tokens.

Parent block, Strict parent block, Not a parent, Always enabled - are common for all rules, see "Common for Styles and Rules".

Group index - integer value that is used to filter text ranges by the syntax tree view. Default is 0, i.e. no filtering performed by the syntax tree.

Change token type - specifies token type which will be assigned to identifier token. This property available only for "Tag detector" rules.

Line break position - specifies position of the line separator (above or under token with Ident Index). This property available only for "Line separator" rules.

Range end condition - reference to the block rule that will close text range, reference to the block rule, text range of which will be closed. This property available only for "Range start" rules.

Range start condition - reference to the block rule text range of that will be closed by this rule. This property available only for "Range end" rules.

Require same ident - When this flag is True, when closing text range condition checks identifiers of range start and range end. Only for "Range end" rules.

Cancel next rules - specifies that block rules checking loop must be broken if this rule is successful.

Text ranges

These properties are applied to created text ranges, i.e. they are used only for rule type "Range start".

Name format string - is used for formatting text range name that is used in TSyntaxTreeView. If it is empty, Identifier token will be taken as the range name. See Name formatting (for block rules) to understand principles of name formatting.

Group format string - is used for formatting text range group name that is used in TSyntaxTreeView. If it is empty, there is no group for this text range. If any text ranges within one parent have identical group name, Group node will be created with caption of this group name and all these text ranges will be children of this group node. See Name formatting (for block rules) to understand principles of building name format.

Collapsed text icon - is used for formatting string which is used to paint collapsed text icon (text displayed when text range is collapsed). When this property is empty collapsed text icon is not displayed (collapsed text range is shown as line separator).

Close block at end of text - when this Flag is True, opened text range will be closed at end of text. Only for "Range start" rules.

Self closing range - specifies whether rule should close previously opened text range before opening new text range.

Initially closed (no end rule) - specifies whether text range should be closed immediately after opening. Text range will occupy tokens that was used when checking conditions or grammar rule.

Display in syntax tree view - specifies whether text range must be displayed in TSyntaxTreeView control. Only for "Range start" rules.

Con not be collapsed - specifies whether text range can be collapsed by user. Only for "Range start" rules.

Draw block staple - specifies whether text ranges of this rule must be pined together by the staple.

Highlight

Style - For "Range start" rule specifies style of all token in the associated text ranges. For "Tag detect" rule specifies style of token with Ident Index. For "Line separator" rule specifies line color of separator.

Highlight type - specifies highlighting region:

Disabled | No highlighting
Only bounding tokens | Only bounding tokens will be highlighted
All tokens without bounds | All tokens without bounds will be highlighted
All range | All tokens in text range will be highlighted

Highlight lines of block - when this flag is True, all lines of text range will be highlighted (will have the same foregound and background colors) with the specified Style. If style have ftBackGround type only backgound will be highlighted. Only for "Range start" rules.

Invert colors of selection - available only if "Highlight lines of block" is True. It specifies style of the selection in Syntax Memo. If it is True, inverted style colors will be used to paint selection, else default selection style will be used. Only for "Range start" rules.

Active highlighting specify that text range may be highlighted only when caret is placed in some predefined position. Only for "Range start" rules.

Caret position - specifies caret position in text range:

Any position (always) | range will be always highlighted independently of caret position
Bounds of the range | range will be highlighted only when caret at the bounds of the range (two caret position) (for example, highlighting brackets)
Bounding tokens | range will be highlighted only when caret at the bounding tokens
In the range | range will be highlighted only when caret is anywhere within text range
Begin of bounding tokens | Text range is highlighted when caret is at the first charecter of the bounding tokens

Select minimal range - specifies that only nearest text range of this rule must be selected for active highlighting. For example, when we highlight "begin ... end" block, we want to highlight only nearest block.

Syntax tree decoration
Here you may specifier styles and image indexes for the tree node associated with text range and group tree node created for this text range.

Conditions

Conditions specify current position validation for this rule. If validating is successful corresponded action defined by the block rule type will be performed. There are any number of condition entries. Each condition have next attributes:

Condition (type) - may be one of the following:

Equal | Token must have token type to be in defined types and token sting must be in key list
Not equal | Token must not have token type to be in defined types and token sting must not be in key list
Mask | Token must have token type to be in defined types and token sting must be in key list (with mask)
Skip | Skips all tokens that have specified token type until next condition will be successful

If key list is empty second part of condition is omitted.

Token types - is a set of token types to check condition.

Key list - is an array of strings to check condition.

Ignore case - specifies whether it compares strings without case sensitive.


Nested lexers

Nested lexers allows use external lexer for the defined text space. For example in HTML we want that script will be highlighted with the corresponded lexer (java script, vb script, perl script, ...).
Sub lexer rules are intended for this purpose.

Because Sub lexer rules are derived from TRuleCollectionItem thay supports common properties:Parent block, Strict parent block, Not a parent, Always enabled. See "Common for Styles and Rules".

Style is applied to all text space where this nested lexer will act. If you want some rules of lexer will be enabled in the text space of nested lexer you must set "Always enabled" property of that rules to TRUE.

Sub lexer combo box allows you to select nested lexer. It containes all lexers with the same Owner, i.e. if you have two lexer on the form, for each of them they will present in sub lexer combo.

Text space of nested lexer detecting

To detect text space two regular expression are used: "Start condition" and "End condition".
Text space will be from the end of the "Start condition" to th start of the end condition.

For example, we include Assembler lexer into Pascal.
Sub lexer = "Pascal"
Start condition = "asm"
End condition = "end"

In this case text space will be from the next character after "asm" to the previous character before "end".

If text space must start from the begin of text set property "Start at the begin of text". Nested lexer starts at the begin of text.

If text space must be closed at end of text set property "End at the end of text". Nested lexer closes at the end of text.

Lexer inheritance

Also if you want to extend some lexer without modifying it, you can do this:
1. Create new lexer.
2. Create "Sub lexer" rule for the base lexer from start to end of text.
3. Create new rules and mark them as "Always enabled".

So you create inherited lexer which have all rules of base lexer plus rules defined in the derived lexer.


Grammar

Grammar rules allows testing complex sequences of tokens.

Grammar rules are separated by the semicolon.

Syntax of grammar definitions:

Rule ::= RuleName '=' Branch_list ';'

Branch_list ::= Item_list
| Branch_list '|' Item_list

Item_list ::= Element
| Item_list Element

Element ::= Item
| Item '*'
| Item '+'
| Item '?'

Item ::= RuleName
| '"' Term '"'
| ''' Term '''
| '<' TokenType '>'
| '(' Item_list ')'

where:
" Term" - case insensitive terminal,
'Term' - case sensitive terminal,
TokenType - name of token type,
RuleName - name of grammar rule.

Repeaters:
'*' - any number of entries including none
'+' - at least one entry
'?' - one entry or none

There are special grammar rule "Skip". It is used after each match operation to skip unused tokens (for example, comments).


Options

Options dialog can be executed by pressing F9 key by default. Or by Right click menu, 'Setup' subitem.
Options are divided to Tabs by categories:
Editor - Configurations of editing text
View - Look and colors
Keys - Hotkeys assignment
Misc - Special functions


Editor configuration

Use the following editor options to control text handling in the Code editor. Check the option to enable it.

Overwrite blocks Replaces a marked block of text with whatever is typed next. If Persistent Blocks is also selected, text you enter is appended following the currently selected block.
Persistent blocks Keeps marked blocks selected even when the cursor is moved using the arrow keys, until a new block is selected.
Double click line Highlights the line when you double-click any character in the line. If disabled, only the selected word is highlighted.
Auto select Selects all text when editor gets focus.
Greedy selection If this option is set selection will contain extra column/line during column/line selection modes.
Keep selection mode Selection enabled for caret movement commands (like in BRIEF).
Word wrap wraps text to multiline, so long line fits on the screen
ReadOnly Disallows editing
Codefolding Display code outline gutter band and allow to hide some portions of text
Keep trailing blanks Keeps any blanks you might have at the end of a line.
Enable text dragging Enables drag&drop operation fo text movement.
Collapse empty lines Collapse empty lines after text range when this rang have been collapsed.
Fixed line height Prevents line height calculation. Line height will be calclculated by means of Default Style.

Auto indent mode Positions the cursor under the first nonblank character of the preceding nonblank line when you press Enter.
Backspace unindents Aligns the insertion point to the previous indentation level (outdents it) when you press Backspace, if the cursor is on the first nonblank character of a line.
Group redo If it is set Redo will involve group of changes.
Group undo Undoes your last editing command as well as any subsequent editing commands of the same type, if you press Alt+Backspace or choose Edit|Undo.
Undo after save Stay undo buffer unchanged after save with SaveToFile method.
Copy to clipboard as RTF Copies selected text also in RTF format.
Smart caretActs on the caret movement (up, down, line start, line end). Caret is moved to the nearest position on the screen.
Keep caret in text Allows move caret only into text like in Memo.
Optimal fill - хз
Disable selection Disables any selection.
Enable column selection Enabled column selection mode.
Float markers If it is set markers are linked to text, so they will move with text during editing. Otherwise they are linked to caret position, and stay unchanged during editing. Also markes save scroll position.
Scroll to last line When it is true you may scroll to last line of text, otherwise you can scroll to last page. When this ption is off and total text height less then client height vertical scroll bar will be hidden.
Break on Right margin Wordwrap will break long lines on Right margin, not the edge of screen
Fixed column move - хз
Variable H scrollbar - Horizontal scrollbar calculates for displaing text, not the full text
Unindent keep align - Stop unindents when unindent will break formatting

Tab width - you can specify width of TAB character and un/indent step in spaces
Tab mode - Specifies action when user press Tab key.
Spaces Inserts spaces to move cursor to the next tab position.
Tab Inserts tab character in text.
SmartTab Inserts spaces to the first non-whitespace character in the preceding line.
Right margin - line will draw after this number of characters


View tab

Interface

Use MDI - Use multidocument interface.
Flat menu style - Use flat styling on menu

Editor

Show separator line - line between folding bar and text field will be drawn
Gradient on gutter - Line numbers will have gradien background from "Line number BG" to "Gutter gradien right"
Highlight links - Links in any text with any lexer will be highlighted. You can Ctrl+Click to open link in browser
Draw current line focus - by default draw border over current line, if lexer doesn't have current line style
Hide selection - Hides selection when editor looses focus
Hide dinamic - Hides dinamic highlighting when editor looses focus
Hide cursor on type - Hides mouse cursor when typing
Show caret in read only - When editor is in read only caret will be shown

Tree

Auto synchronize - Select corresponding item in tree when moving in text
Auto collapse - Tree will autocollapse old item when open new
Auto expand - Autoexpand items when click on it (else need doubleclick)
Sort items - Items will be sorted in ascending order
Hot track - Hilight item under mouse cursor


Keys tab

You can assign multiple keystrokes to one action. All actions divided to groups.
Use Hotkey field - to capture keystroke
Set button - to save keystroke to selected action
Clear button - to clead selected keystroke of action
// econtrol.ru // sepa.spb.ru //


Misc tab

File State History - How much last opened files remember. (Position, selection, bookmarks, encoding)
Search/Replace History - Remember this number of last Search/Replace queries

Editor

Ask to overwrite ReadOnly - Will ask if you save ReadOnly file. If not checked, then save without confirmation
Bring TC to top on ESC - If checked, then TotalCommander window will be active after you close one frame by ESC. Else Syn2 window will be active (on top). Useful when Use MDI enabled
Remember last window size/position - Save Syn2 window position, and open all new windows in same place.
Use internat HTML autocomplete- If cursor stay in text colored by 'CSS' or 'subHTML' lexer, then you will get inplace HTML tag/attribute autocomplete by pressing Ctrl+Space

AutoComplete Popup behavior:
No filtration - when you type function name, no list filtering will occure
Start of string - filter all functions that not start as entered text
Any part of string - filter all functions that not contain entered text
Characters are in the same order - Entered characters exists in same ordes (For example "WaitForCalc" will be found by "wolc" substring)

If File was changed outside:
Do nothing - no action
Ask to Reload/Save - You will be asked to save file if it's modified and to Reload changes in file.
Reopen without asking (Log mode) - File will be reopened without asking (Only ask to Save if modified). If you view not beginning of file, after reloding you will be scrolled automatically to end of file. (So if log grows up - you always see beginning of file, if log grows down - you'll autoscrolled to end of file) File reloads in ReadOnly mode by default.

Chm file for context help- You can use context help (F1) by default. When you press hotkey - curent word under cursor, or selected text replace %WORD% variable. For official PHP chm help downloaded from (http://www.php.net/download-docs.php) you can use this settings:
C:\path\to\php_manual_en.chm::en/function.%WORD%.html - for english
C:\path\to\php_manual_ru.chm::ru/function.%WORD%.html - for russian
etc.

// econtrol.ru // sepa.spb.ru //


Statusbar

Description of fields (left to right)
1. Cursor position (X:Y) and selection (length/lines)
2. Current encoding. If you don't modify file ('Save' button is gray) then change encoding will reopen file in selected encoding. If you already modify file, then encoding change will not reload file. Selected encoding will be used for Save.
So, for convert file from KOI8 encoding to Unicode (for example): Open file, select KOI8 encoding, write elsewhere space and delete it, Save button now active, choose Unicode encoding and save file.
There is autodetection of encoding only for UTF8, Unicode, UnicodeBE, all other files opens as Ansi.
3. Line endings (used for saving, automatically detected on open file)
4. Current Highlighter. You can manage active highlighter list by 'Manage Library' menu item. All lexers saved in 'HL\LexLib.lxl' file
5. Char at cursor info ("char at cursor" #dec hex code of this char)
6. ReadOnly indicator
7. Hints and messages field
// econtrol.ru // sepa.spb.ru //


Search/Replace

Incremental search:

Default hotkey is Ctrl+E When you in incremental search mode, statusbar hides all panels and displays your query. You can press any non-character key (Enter, Esc) to leave incremental search mode. When word is found, it's easy to find next/prev instance by Ctrl+Shift+Upor Ctrl+Shift+DownThere are options MatchCase and WholeWords used from Search/Replace dialog.

QuickSearch:

When you press default hotkey Ctrl-Ffocus set to QuickSearch field.
This field works as incremental search, but there also works F3/Enter and Shift-F3 combinations to search next/prev. Use Tab/Escto leave QuickSearch field. History of this field is saved by default (20 last queries). You can use Up/Down to select item and press Enter to search it.
Additional options MatchCase, WholeWords, RegExp used from Search/Replace dialog.

Search/Replace:

Default hotkey is Ctrl-R This dialog allows you to Find more complex queries. Options are self-explained. By default there is history for Search and Replace fields 20 items. This can be changed in Options at Misc tab.


About

Components used:

Econtrol Syntax Editor
TBX
Toolbar2000
FastCode/Move/MM
JVCL 		http://econtrol.ru/
http://g32.org/tbx/
http://jrsoftware.org/
http://fastcodeproject.org/
http://jcl.sf.net

Thanx:

Michael
Atorg
 http://econtrol.ru/
http://atorg.net.ru/

Links:

Last version
Debug
Engine/Lexers/Help 		http://sepa.spb.ru/?syn2
http://ghisler.ch/board/viewforum.php?f=6
http://econtrol.ru/download_e.html